Method and Device for the Control of Functions in a Vehicle Using Gestures Performed in Three-Dimensional Space, and Related Computer Program Product

ABSTRACT

A method to control functions in a vehicle using gestures carried out in three-dimensional space. When it is determined that a first gesture carried out in three-dimensional space is detected by an image-based detection procedure, it is determined whether the first gesture is a gesture directed towards a virtual object superimposing a real environment around the vehicle. If it is determined that the first gesture has been detected, it is determined whether a second gesture carried out in three-dimensional space is detected by the image-based detection procedure. If so, it is determined whether the detected second gesture is a gesture allocated to a manipulation of the virtual object, and the virtual object is manipulated accordingly.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to PCT Application No. PCT/EP2012/005080,filed Dec. 8, 2012, a National Stage application of which is U.S.application Ser. No. ______ (Attorney Docket No. 095309.66677US), andPCT Application No. PCT/EP2012/005081, filed Dec. 8, 2012, a NationalStage application of which is U.S. application Ser. No. ______ (AttorneyDocket No. 095309.66616US).

BACKGROUND AND SUMMARY OF THE INVENTION

Exemplary embodiments of the present invention relate to a method anddevice to control functions in a vehicle using gestures carried out inthree-dimensional space as well as a relevant computer program product.

U.S. patent publication US 2008/0065291 A1 discloses a method and adevice to control functions in a vehicle using gestures carried out inthree-dimensional space, in which it is determined whether a gesturecarried out in three-dimensional space is detected by means of animage-based detection procedure or not; it is determined whether thedetected gesture is a gesture allocated to an operation of a function ornot and the function is operated in the case that it is determined thatthe detected gesture is the gesture allocated to the operation of thefunction.

Exemplary embodiments of the present invention are directed to a method,a device and a relevant computer program product, which allow agesture-based control in a vehicle for a user to interact with a realenvironment around the vehicle in a simple and reliable way.

According to a first aspect, a method to control functions in a vehicleusing gestures carried out in three-dimensional space feature a) adetermination of whether a first gesture carried out inthree-dimensional space is detected by means of an image-based detectionprocedure or not, b) a determination of whether the first gesture is agesture that is directed towards a virtual object superimposed on a realenvironment around the vehicle or not, in the case that it is determinedthat the first gesture has been detected, c) a determination of whethera second gesture carried out in three-dimensional space is detected bymeans of the image-based detection procedure or not, d) a determinationof whether the detected second gesture is a gesture allocated to amanipulation of the virtual object or not, in the case that it isdetermined that the second gesture has been detected, and e) amanipulation of the virtual object, in the case that it has beendetermined that the detected first gesture is the gesture that isdirected towards the virtual object superimposed on the real environmentaround the vehicle and in the case that the detected second gesture isthe gesture that is allocated to the manipulation of the virtual object.

According to one embodiment, the manipulation of the virtual objectfeatures a displacement of the virtual object onto a display unit of thevehicle.

According to a further embodiment, the manipulation of the virtualobject features a copying of the virtual object onto a display unit ofthe vehicle.

According to a further embodiment, the manipulation of the virtualobject features a depiction of information regarding the virtual objecton the display unit of the vehicle.

According to a further embodiment, the superimposing of the virtualobject onto the real environment around the vehicle is carried out bymeans of a head-up display and the display unit is at least oneinstrument panel and at least one central display unit.

According to a further embodiment, the superimposing of the virtualobject onto the real environment around the vehicle is carried out bymeans of a projection of the virtual object into the real environmentaround the vehicle and the display unit is at least one instrument paneland at least one central display unit.

According to a further embodiment, the virtual object is superimposedonto the real environment around the vehicle at a position correspondingto an object present in the real environment, which is allocated to thevirtual object.

According to a further embodiment, the image-based detection procedureis camera-based and a position of an object carrying out a gesture inthree-dimensional space is detected.

According to a second aspect, a device to control functions in a vehicleusing gestures carried out in three-dimensional space has equipment,which is designed to carry out the method described above or theembodiments thereof.

According to a third aspect, a computer program product to controlfunctions in a vehicle using gestures carried out in three-dimensionalspace is designed to carry out the method described above or theembodiments thereof directly in combination with a computer or acomputer system or indirectly after carrying out a pre-determinedroutine.

According to the first to third aspects and their embodiments, a usercan display information about relevant virtual objects by means ofvirtual objects superimposed on the real environment around the vehicleby means of a gesture operation on the display unit of the vehicle andin this way can interact with the real environment around the vehicle.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The present invention is explained in more detail below by means of anexemplary embodiment with reference to the enclosed drawing.

In the drawing is shown:

FIG. 1 a schematic depiction of a basic structure of a display unit anda detection concept according to an exemplary embodiment of the presentinvention.

FIG. 2 a further schematic depiction of the basic structure of thedisplay unit and the detection concept according to the exemplaryembodiment of the present invention.

FIG. 3 a schematic depiction of the basic structure of the display unitand an installation location of a detection device in an overheadcontrol unit according to the exemplary embodiment of the presentinvention.

FIG. 4 a schematic depiction of the basic structure of the display unitand an installation location of a detection device in an inner mirroraccording to the exemplary embodiment of the present invention.

FIG. 5 a schematic depiction of a real environment around the vehiclewith virtual superimposed objects according to the exemplary embodimentof the present invention; and

FIG. 6 a flow diagram of a method to control functions in a vehicleusing gestures carried out in three-dimensional space according to theexemplary embodiment of the present invention.

DETAILED DESCRIPTION

The description of one exemplary embodiment of the present inventionoccurs below.

It is to be noted that it is hereinafter assumed that a display unit isa preferably central display or an instrument panel of a vehicle,preferably a motor vehicle, and a method to control functions depictedon the display unit using gestures carried out in three-dimensionalspace in the vehicle is carried out.

Furthermore, a gesture described below is a gesture carried out inthree-dimensional space by a user of the vehicle by means of a hand or afinger of the user, without touching a display, such as, for example, atouch screen, or a control element, such as, for example, a touch pad.

The image-based capturing device described below can be any expedientcamera, which is able to detect a gesture in three-dimensional space,such as, for example, a depth camera, a camera having structured light,a stereo camera, a camera based on time-of-flight technology or aninfra-red camera combined with a mono camera. A plurality of anycombinations of such cameras is possible. An infra-red camera combinedwith a mono-camera improves a detection capability, as a mono camerahaving a high image resolution additionally provides intensityinformation, which offers advantages during a background segmentation,and a mono camera is impervious to extraneous light.

FIG. 1 shows a schematic depiction of a basic structure of a displayunit and a detection concept according to an exemplary embodiment of thepresent invention.

In FIG. 1, the reference numeral 10 refers to a display unit of avehicle, the reference numeral 20 refers to a valid detection region ofan image-based detection device, the reference numeral 30 refers to anoverhead control unit of the vehicle, the reference numeral 40 refers toan inner mirror of the vehicle, the reference numeral 50 refers to acentral console of the vehicle and the reference numeral 60 refers to adome of the vehicle.

The basic control concept is that a gesture operation to controlfunctions by means of a gesture carried out by a hand or a finger of auser in the valid detection region 20 is carried out inthree-dimensional space if the gesture carried out is detected as apredetermined gesture in the detection region 20 by means of theimage-based detection device.

The valid detection region 20 is determined by an image-based detectiondevice, which is able to detect a three-dimensional position of the handor the fingers of the user in the three-dimensional space. Preferablythe image-based detection device is a depth camera integrated into thevehicle.

The image-based detection device must be integrated such that a gestureoperation is allowed by a relaxed hand and/or arm position of the userat any position in the region above the dome 60 and the central console50 of the vehicle. Thus a valid detection region can be limited fromabove by an upper edge of the display unit 10 and from below by aminimum distance to the dome 60 and the central console 50.

A gesture operation is activated if a first gesture is detected in thevalid detection region 20, which is a first predetermined gesture. Thefirst predetermined gesture is a static gesture carried out by movingthe hand or the finger of the user into the valid detection region 20and subsequently temporarily leaving the hand or the finger of the userin the valid detection region 20 for a first predetermined amount oftime.

The gesture operation is deactivated by moving the hand or the finger ofthe user out of the valid detection region. A laying of the hand or thearm of the user on the central console 20 and a control of components ofthe vehicle is carried out under the valid detection region 20, wherebya gesture operation is not activated.

A static gesture is not carried out in the case of a gesticulation inthe vehicle and in the case of moving the hand or the finger of the userto a control element, whereby a gesture operation is not activated.

FIG. 2 shows a further schematic depiction of the basic structure of thedisplay unit and the detection concept according to the exemplaryembodiment of the present invention.

In FIG. 2, the same reference numerals refer to the same elements as inFIG. 1 and the reference numeral 70 refers to an item present in or onthe central console 50 as an obstructive object, such as, for example, adrink container in a cup holder.

The statements made above with regard to FIG. 1 likewise apply for FIG.2.

A lower boundary of the valid detection region 20 is dynamically adaptedto the item 70. Such a context-dependent adaptation of the validdetection region as an interaction region is carried out such that adepth contour of the valid detection region is carried out by means ofdepth information of the image-based detection device, such as, forexample, a depth camera, in real time in the case of a detection of agesture. This means that a valid gesture must be carried out above theitem 70.

An arrangement of the image-based detection device in an overhead regionof the vehicle leads to the following advantages: No sunlight shinesinto a lens of the image-based detection device. A complete detectionregion is also covered in an adjacent region of the display unit 10 as avalid detection region 20. There is a high image resolution in the maininteraction directions to the left, to the right, in front of and behindthe gesture operation. The image-based detection device is made up of anormal visual range of driver and passenger. Overhead components can beeasily standardized for different series with few design variations. Fewrequirements for a detection distance are required.

With respect to FIG. 3 and FIG. 4, two possible installation locationsfor the image-based detection device in the overhead region of thevehicle are illustrated.

FIG. 3 shows a schematic depiction of the basic structure of the displayunit and an installation location of the detection device in theoverhead control unit according to the exemplary embodiment of thepresent invention.

In FIG. 3 the same reference numerals refer to the same elements as inFIG. 1 and FIG. 2 and the reference numeral 100 refers to a maximumdetection angle of an image-based detection device integrated into theoverhead control unit 30 of the vehicle.

The statements made above with regard to FIG. 1 and FIG. 2 likewiseapply for FIG. 3.

As can be seen in FIG. 3, the complete valid detection region 20 iscovered with the image-based detection device integrated into theoverhead control unit 30. A further advantage of the image-baseddetection device integrated into the overhead control unit 30 is thatthe greatest possible vertical distance to the valid detection region 20is achieved.

FIG. 4 shows a schematic depiction of the basic structure of the displayunit and an installation location of a detection device in an innermirror according to the exemplary embodiment of the present invention.

In FIG. 4 the same reference numerals refer to the same elements as inFIG. 1 and FIG. 2 and the reference numeral 110 refers to a maximumdetection angle of an image-based detection device integrated into theinner mirror 40 of the vehicle.

The statements made above with regard to FIG. 1 and FIG. 2 likewiseapply for FIG. 4.

As can be seen in FIG. 4, the complete valid detection region 20 iscovered with the image-based detection device integrated into theoverhead control unit 30. In order to compensate for a changingalignment of the image-based detection device due to an adjustment ofthe inner mirror 40, an alignment offset of the image-based detectiondevice is corrected by means of a contour of the central console 50 inorder to carry out a position calibration.

FIG. 5 shows a schematic depiction of a real environment around thevehicle with virtual superimposed objects according to the exemplaryembodiment of the present invention.

In FIG. 5 the reference numeral 200 refers to a windscreen of a vehicle,the reference numeral 300 refers to a real environment around thevehicle, the reference numeral 400 refers to a first virtual object, thereference numeral 500 refers to a second virtual object, the referencenumeral 600 refers to a third virtual object, the reference numeral 700refers to a instrument panel, the refers numeral 800 refers to asteering wheel and the reference numeral 900 refers to a central displayunit.

Virtual objects, which have a connection to an object present in thereal environment 300, which is located at this position, aresuperimposed onto a real environment around the vehicle, such as, forexample, the real environment 300 in front of the vehicle that isvisible through the windscreen 200, as is indicated schematically bymeans of buildings, a road and a proceeding vehicle in FIG. 5. Forexample, the virtual object can be the first virtual object 400allocated to a car park, which is superimposed at a position on the realenvironment 300, at which the car park allocated to the virtual object400 is present in the real environment, the virtual object can be thesecond virtual object 500 allocated to a public toilet, which issuperimposed at a position on the real environment 300 at which thepublic toilet allocated to the virtual object 500 is present in the realenvironment 300, and/or the virtual object can be the third virtualobject 400 allocated to a restaurant, which is superimposed at positionin the real environment 300 at which the restaurant allocated to thevirtual object is present in the real environment 300. Any furtherand/or other types of objects and virtual objects are likewise possible.

Superimposing of the virtual objects onto the real environment 300 can,for example, be carried out by means of a display of the virtual objectson a so-called head-up display on the windscreen 200 of the vehicle. Afurther possibility to carry out the superimposing of the virtualobjects onto the real environment 300 can be implemented, for example,by means of a projection of the virtual objects onto the realenvironment 300 by means of a suitable projection device.

The allocation of virtual objects to objects present in the realenvironment 300 is carried out, for example, on the basis of positioncoordinates, such as, for example, GPS coordinates, which are obtainedby means of a navigation system in the vehicle, and/or an objectrecognition of the objects present in the real environment 300.

Information that features additional or detailed information with regardto the objects in the real environment 300 relating to the virtualobjects is associated with the objects superimposed onto the realenvironment 300. For example, information regarding the first virtualobject 400 can feature a price for the car park present in the realenvironment 300 or information regarding the third virtual object 600can feature a menu for the restaurant present in the real environment300.

The virtual object can be marked or selected by means of a gesture by auser, which is directed towards a virtual object, and the virtual objectcan be moved or copied to the instrument panel 700 or to the centraldisplay unit 900 by means of a further gesture, which is a dynamicgesture or a swiping movement towards the instrument panel 700, which islocated behind the steering wheel 800, or the central display unit 900,and information regarding the virtual object can be depicted on theinstrument panel 700 or the central display unit 900.

FIG. 6 shows a flow diagram of a method to control functions in avehicle using gestures carried out in three-dimensional space accordingto the exemplary embodiment of the present invention.

It is to be noted that a process flow of the flow diagram in FIG. 6 isswitched on, for example, after an initialization point, such as, forexample, after switching on an ignition of the vehicle, and is carriedout in repeating cycles until an end point, such as, for example, aswitching-off of the ignition of the vehicle, is reached. Alternatively,the initialization point can, for example, be the point in time ofstarting a motor of the vehicle and/or the end point can be the point intime of switching off the motor of the vehicle. Other initializationpoints and end points are likewise possible according to the presentapplication.

A distinction can be made as to whether a gesture is carried out by adriver or by a passenger, which is particularly advantageous in aso-called split view display, which is able to display different piecesof information to the driver and the passenger simultaneously. Likewise,the distinction as to whether a gesture is carried out by a driver or bya passenger out is advantageous with regard to an ergonomic control bythe driver or the passenger.

Below, it is assumed that the detected gesture can be both a gesturecarried out by the driver and a gesture carried out by the passenger.

Furthermore, it is to be noted that in the case of the distinctiondescribed above between a gesture of the driver and of the passenger,the method of the flow diagram in FIG. 6 is carried out both for thedriver's side and for the passenger's side. The process sequence shownin FIG. 6 can be carried out expediently, for example, in parallel, inseries or in a connected manner for the driver's side and thepassenger's side.

In step S100 it is determined whether a first gesture is detected ornot. In the case that the first gesture is not detected (“No” in stepS100), the process sequence returns to step S100. In the case that thefirst gesture is detected (“Yes” in step S100), the process sequenceadvances to step S200.

In step S200 it is determined whether the detected first gesture is agesture which is directed towards the virtual object superimposed ontothe real environment around the vehicle or not. In the case that thefirst gesture is not a gesture which is directed towards the virtualobject superimposed onto the real environment around the vehicle (“No”in step S200), the process sequence returns to step S100. In the casethat the first gesture is a gesture which is directed towards thevirtual object superimposed onto the real environment around the vehicle(“Yes” in step S200), the process sequence advances to step S300.

The gesture, which is directed towards the virtual object superimposedon the real environment around the vehicle, is, for example, a firstpredetermined gesture, which is directed statically towards the virtualobject for a first predetermined period of time in an interaction regionin three-dimensional space. The first predetermined gesture is detected,as has been described above with reference to FIGS. 1 to 5. Theinteraction region corresponds to the valid detection region describedabove.

In step S300, the virtual object is marked or selected. After step S300,the process sequence advances to step S400.

In step S400 it is determined whether a predetermined abort condition isfulfilled or not. In the case that the predetermined abort condition isfulfilled (“Yes” in step S400), the process sequence returns to stepS100. In the case that the abort condition is not fulfilled (“No” instep S400), the process sequence advances to step S500.

The predetermined abort condition can, for example, be that no gesturehas been detected for a fourth predetermined period of time.

In step S500 it is determined whether a second gesture is detected ornot. In the case that the second gesture is not detected (“No” in stepS500), the process sequence returns to step S500. In the case that thesecond gesture is detected (“Yes” in step S500), the process sequenceadvances to step S600.

In step S600 it is determined whether the detected second gesture is agesture allocated to a manipulation of the virtual object or not. In thecase that the second gesture is not a gesture allocated to themanipulation of the virtual object (“No” in step S600), the processsequence returns to step S500. In the case that the second gesture is agesture allocated to the manipulation of the virtual object (“Yes” instep S600), the process sequence advances to step S700.

The gesture allocated to the manipulation of the virtual object is asecond predetermined gesture, which is dynamic in the interaction regionin three-dimensional space and is directed from the virtual object thatsuperimposes the real environment, is marked, or is selected, towards adisplay unit, such as, for example, the instrument panel 700 or thecentral display unit 900.

In step S700, the virtual object is manipulated, for example it is movedor copied to the display unit or information relating to the marked orselected object is displayed on the display unit. After step S700, theprocess sequence returns to step S700.

The method described above can be carried out by means of equipment,which forms a device to control functions in a vehicle. A display unitis preferably a central display of the vehicle, preferably of a motorvehicle.

Although specific installation locations for respective cameras areshown in FIGS. 3 and 4, respective cameras can be arranged in otherexpedient installation locations.

A simple and quick controllability is implemented by the image-basedgesture control described above, which improves an control comfort, acontrol flexibility and control experience for the user, significantlyincreases the freedom of design for a vehicle interior and allows aninteraction of a user with the real environment.

The exemplary embodiment described above is able to be implemented as acomputer program product, such as, for example, a storage medium, whichis designed to carry out a method according to the exemplary embodimentabove, interacting with a computer or several computers, i.e. computersystems, or other processing units. The computer program product can bedesigned such that the method is carried out only after theimplementation of a predetermined routine, such as, for example, aset-up routine.

Although the present invention has been described above by means of anexemplary embodiment, it is to be understood that different embodimentsand changes can be carried out without leaving the scope of the presentinvention, as is defined in the enclosed claims.

The disclosure of the drawing is exclusively referred to regardingfurther features and advantages of the present invention.

1-10. (canceled)
 11. A method to control functions in a vehicle usinggestures carried out in three-dimensional space, the method comprising:a) determining whether a first gesture carried out in three-dimensionalspace is detected by an image-based detection procedure; b) determining,if is determined that the first gesture has been detected, whether thefirst gesture is a gesture directed towards a virtual objectsuperimposed onto a real environment around the vehicle; c) determiningwhether a second gesture carried out in three-dimensional space isdetected by the image-based detection procedure; d) determining, if itis determined that the second gesture has been detected, whether thedetected second gesture is a gesture allocated to a manipulation of thevirtual object; and e) manipulating the virtual object if it has beendetermined that the detected first gesture is the gesture directedtowards the virtual object and the detected second gesture is thegesture allocated to the manipulation of the virtual object.
 12. Themethod of claim 11, wherein the manipulation of the virtual objectinvolves moving the virtual object onto a display unit of the vehicle.13. The method of claim 11, wherein the manipulation of the virtualobject involves copying of the virtual object onto a display unit of thevehicle.
 14. The method of claim 11, wherein the manipulation of thevirtual object involves depicting information relating to the virtualobject on a display unit of the vehicle.
 15. The method of claim 12,wherein the superimposing of the virtual object onto the realenvironment around the vehicle is performed by a head-up display and thedisplay unit is at least one instrument panel and at least one centraldisplay unit.
 16. The method of claim 12, wherein the superimposing ofthe virtual object onto the real environment around the vehicle isperformed by projecting the virtual object into the real environmentaround the vehicle and the display unit is at least one instrument paneland at least one central display unit.
 17. The method of claim 11,wherein the virtual object is superimposed onto the real environmentaround the vehicle at a position corresponding to an object present inthe real environment, which is assigned to the virtual object.
 18. Themethod of claim 11, wherein the image-based detection procedure iscamera-based and a position of an object carrying out a gesture isdetected in three-dimensional space.
 19. A device to control functionsin a vehicle using gestures carried out in three-dimensional space,wherein the device is configured to: a) determine whether a firstgesture carried out in three-dimensional space is detected by animage-based detection procedure; b) determine, if is determined that thefirst gesture has been detected, whether the first gesture is a gesturedirected towards a virtual object superimposed onto a real environmentaround the vehicle; c) determine whether a second gesture carried out inthree-dimensional space is detected by the image-based detectionprocedure; d) determine, if it is determined that the second gesture hasbeen detected, whether the detected second gesture is a gestureallocated to a manipulation of the virtual object; and e) manipulate thevirtual object if it has been determined that the detected first gestureis the gesture directed towards the virtual object and the detectedsecond gesture is the gesture allocated to the manipulation of thevirtual object.
 20. A non-transitory computer-readable medium controlfunctions displayed on a display unit of a vehicle using gesturescarried out in three-dimensional space, wherein the computer-readablecontains instructions, which when executed by a device, cause the deviceto: a) determine whether a first gesture carried out inthree-dimensional space is detected by an image-based detectionprocedure; b) determine, if is determined that the first gesture hasbeen detected, whether the first gesture is a gesture directed towards avirtual object superimposed onto a real environment around the vehicle;c) determine whether a second gesture carried out in three-dimensionalspace is detected by the image-based detection procedure; d) determine,if it is determined that the second gesture has been detected, whetherthe detected second gesture is a gesture allocated to a manipulation ofthe virtual object; and e) manipulate the virtual object if it has beendetermined that the detected first gesture is the gesture directedtowards the virtual object and the detected second gesture is thegesture allocated to the manipulation of the virtual object.